DOCUMENT RESUME 



ED 431 430 



IR 057 392 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Adams , Carl 

The Skills Audit Approach To Facilitate Undergraduate 
Learning . 

1998-12-00 

7p.; In: Proceedings of the International Academy for 
Information Management (lAIM) Annual Conference (13th, 
Helsinki, Finland, December 11-13, 1998); see IR 057 374. 
Reports - Evaluative (142) -- Speeches/Meeting Papers (150) 
MFOl/PCOl Plus Postage . 

Audits (Verification) ; *Business Administration Education; 
Case Studies; Distance Education; Educational Change; 
Feedback; Foreign Countries; *Formative Evaluation; Higher 
Education; Information Technology; Management Information 
Systems; *Self Evaluation (Individuals); *Skill Analysis; 
♦Student Evaluation; *Undergraduate Study 
England 



ABSTRACT 



Changes in the United Kingdom higher education system have 
resulted in a lack of formative (i.e., student feedback) assessment: 
increased student numbers and reduced funding resulting in bigger classes 
with less tutor contact, as well as a more diverse set of student entrants 
with a wider set of skills resulting in a greater need for formative 
assessment. It has become increasingly difficult in the information 
systems/information technology field to monitor individual student 
performance and give individual feedback and direction. This paper examines 
one mechanism to address this difficulty, a skills audit approach. The use of 
a skills audit approach is described for three units at the Southampton 
Business School (England) ; the units are Information Systems and Business 
Skills, both core level 1 units, and Computer Auditing, an optional level 3 
unit. The pedagogic value of the skills audit approach for different levels 
of study is discussed. A self audit approach seems particularly relevant for 
meeting the formative assessment needs of students and encouraging self study 
and ownership. There are clear indications of the suitability of such an 
approach for distance learning. ( Author /MES) 
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Changes in the UK education system have resulted in a lack of formative (i.e., student feedback) assessment: 
increased student numbers and reduced funding resulting in bigger student classes with less tutor contact; also, 
^ a more diverse set of student entrants with a wider set of skills, resulting in a bigger need formative assessment. 

^ It has become increasingly difficult in the IS/IT field to monitor individual student performance and give 

^ individual feedback and direction. This paper examines one mechanism to address this difficulty, a skills audit 

^ approach. The use of a skills audit approach is described for three units, two at level 1 and one at level 3 of 

Q undergraduate study. Discussion is given on the pedagogic value of the skills audit approach for different level 

of study. A self audit approach seems particularly relevant for meeting the formative assessment needs of 
students and encouraging self study and ownership. There are clear indications of the suitability of such an 
approach for distance learning. 







INTRODUCTION AND BACKGROUND 

The expansion of Higher Education (HE) in the UK over 
the last decade, coupled with the reduction in funding per 
student, has had many effects on the teaching and learning 
process. Notably, larger class sizes, increased competition 
for books and study resources, reduced access to tutorial 
support from faculty and less detailed or frequent 
feedback on progress and course-work (Gibbs, Lucas and 
Simonite 1996, p261). These changes have been further 
exasperated by the increased variety of entry routes, 
including NVQs (National Vocational Qualifications), 
GNVQs (General National Vocational Qualifications), 
mature students as well as the traditional A* levels 
(Hyland 1994; Burke 1995), with each of these routes 
fostering and developing different skill sets in students. 
This greater variety of entrant skills to HE increases the 
need for individual feedback and support to students, 
however, as already highlighted, increased student 
numbers and reduced fimds per student have resulted in 
less detailed and frequent feedback and faculty support. 

The need for individual feedback on student progress is 
probably more pronounced in Information Systems/ 
Information Technology (IS/IT) related courses and units 
where further influences have taken place. The topic areas 
are in a continual state of flux with the depth and breadth 



of technology related issues increasing. The skill base of 
entrants to HE are changing with greater computer 
literacy for some students and more to learn for the non 
computer literate. The destination of graduates, the 
business environment, is also in a state of flux with new 
IT related business practices and further reliance on IT. 
Additionally there has been a wider integration of 
technology into the teaching and learning process 
(Grandgenett et al 1997; Kapur and Stillman 1997). 

From a teaching perspective it has become increasingly 
difficult in the IS/IT field to monitor individual student 
performance and give individual feedback and direction. 

This paper examines one mechanism to address this 
inconsistency by using a Skills Audit to provide individual 
feedback and monitoring. In addition the Skills Audit 
techniques promote independent learning skills. Student 
learning requirements change over levels of study, for 
instance level 3 students are typically more self reliant in 
their studies (Adams and King 1 994). This paper describes 
the use of a Skills Audit on three units offered on Business 
courses, two of the units being core level 1 units, while 
the other is an elective unit. 

The rest of this paper is structure as follows. First the 
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individual cases of using skills audit are described. The 
paper then discusses the pedagogic justification of such an 
approach with discussion of how it may be applied at 
different levels of study and different modes of study, 
including distance learning. Finally, the salient points of 
skills audit for each level of study are summarised along 
with a discussion of its use as a distance learning tool. 

SKILLS AUDIT: THREE CASES 

Skills Audit has been used on three units at the 
Southampton Business School. The units are Information 
Systems and Business Skills, both core level 1 units, and 
Computer Auditing, a optional level 3 unit. All the 
students on the units are studying business related degrees. 
For each of these units a 1 5 week semester system is used, 
consisting of 12 weeks taught classes with revision and 
exams in the final weeks. 

A skills audit approach has been used differently in each 
of these three cases. The differences apply to wlio are the 
main recipients of the audit and what they do with the 
results. The differences also relate to the level of study 
and the associated learning requires, e.g. level 3 students 
are typically more independent learners. 

The following is a description of each of the cases. 

Level 1 Unit: Business Skills 

On the Business Schools Undergraduate Programme 
(BSUP), all students take the Business Skills unit which 
incorporates maths, IT and literacy. A skills audit was 
used on the maths element of the unit and comprised of 
new entrants taking a computer marked diagnostic maths 
test in induction week. 

The background of the unit and the move towards a skills 
audit approach, centre on the main problems encountered 
due to the wide variety of skills for new entrants. For 
business related degrees, a wide variety of entry 
qualifications were accepted. So typically tutorial groups 
would include students with maths skills ranging from 
good Advanced (A*) Level mathematics to mature 
students with little or no formal qualifications in maths. 
This resulted in tutorial material being not relevant to 
many students: the students with good maths skills found 
the sessions too easy, while students with poor maths 
skills found the sessions too difficult. 

What was needed was a mechanism to identify the maths 



skills of individual students, give individual feedback to 
students and to enable further guidance to be given to the 
less able students. The result was developing a computer 
marked maths diagnostic test which each student took in 
induction week. Individual feedback was then given to 
students within a week of the test. The feedback consisted 
of an overall mark, and more importantly, an indication of 
how well they performed in four key areas of maths, these 
being Arithmetic Skills, Estimating Skills, Averaging 
Skills and Algebraic Skills. In addition, it was indicated 
where students were particularly week and should seek 
further support from the Study Assistance who deal with 
remedial work. A self study learning pack was developed 
covering the numeracy items being tested. Students that 
performed sufficiently well on the diagnostic test were 
APL*ed from the numeracy element of the Business Skills 
unit, enabling them to devote more attention on the other 
elements of the skills unit. 

The overall result of the approach was more efficient use 
of staff and student resources. Students had better 
feedback on where their strengths and weaknesses lie, and 
on where to direct their study. Staff were able to identify 
weaker ^dents earlier and direct them to Study 
Assistance. Also staff in the tutorial sessions were able to 
concentrate on problem areas since the self study material 
enabled students to work at their own pace. 

The approach has developed over the last three years with 
some minor modifications each year. The skills team are 
currently fairly happy with the existing structure, in 
addition all the student performance is recorded in a 
database in which staff can monitor the overall skill-set of 
new entrants from year to year. 



Level 1 Unit: Information Systems 

The Information Systems unit covered current technology, 
organisational aspects, and systems theory. The original 
assessment for this unit was a written report assignment, 
usually towards the end of the semester, with an end of 
semester exam. This proved inappropriate for identifying 
weaker students early on, and feedback to students was 
typically too late. As with the skills unit another main 
problem was the wide variety of knowledge and skills of 
the new entrants, typically, with some student having high 
level of practical (though selective) IT skills gained in a 
working environment, along students with very low IT 
skills. 

A different approach to skills audit was taken on this unit. 
The audit consisted of two short tests during the semester. 
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one in week 5 the other in week 8. The tests entailed a 
bank of multi-choice questions and an essay question. The 
first test covered mainly technology issues, while the 
second test covered mainly organisational issues. The test 
scripts were handed back to students with appropriate 
comments and correct answers. Typically the scripts were 
handed back in the week following the test. The use of 
multi-choice questions facilitated quicker marking, though 
more effort was needed in developing the bank of 
questions. This was an audit after students had studied a 
topic area, so effectively auditing the effectiveness of 
students studying in a topic area. The results were 
monitored by the unit leader. This enabled the 
identification of weaker students earlier on and, enabling 
further direction to be given in preparation for the 
following test and the fmal end of semester exam. Also, 
it enabled overall areas of weakness to be identified, 
which could then be addressed in the following lectures 
and be given greater attention in the following year. The 
multi-choice questions we used as an indication of the 
breadth of background reading done by the students, 
where as the essay was used to show the depth of study 
undertaken. 

Overall the students were fairly positive to the assessment 
me^od, liking particularly the quick feedback and the 
mix of multi-choice and essay questions. 

Level 3 Unit, Computer Auditing 

The Computer Auditing unit covers security, 
management, legal and flnancial auditing issues of using 
technology in business. The students were from different 
courses including Accountancy, Business Studies and 
direct entrants into level 3 from HND courses. 
Effectively, the students taking the unit had a variety of 
different skills sets. The problem I faced here was "/low 
do I give individual feedback covering understanding in 
a wide set of issues, to 40 students, which I see only a few 
hours a week, and who will be at different levels in each 
of the topics?'\ 

A skills audit approach was adopted to meet the formative 
assessment needs of these students. The approach differed 
to that used with level one students, by incorporating 
more self assessment and peer review. At the start of each 
topic area a list of items to be covered was produced, an 
example is shown in figure 1, which covers the legal 
aspects. Students were then asked to go through the list, 
individually, and identify how much they knew about 
each item. Students then worked in small groups going 
through the list and discussing each item. The discussion 
included items such as: what constitutes "Very low”. 



"Medium” and "High” levels of understanding/ 
knowledge?; and where to find information about the 
topic. Each group generated a list of three items that the 
group knew most about and the three items that the group 
knew least about. These were then fed back to the other 
students and tutor. The only feedback to the tutor from 
these lists were the three most and least known about 
topic areas. This enabled some modification to the 
delivery and content of the topic material (particularly 
references) to meet the needs of a particular cohort. 

Students kept their own lists so they could see where to 
concentrate their efforts and monitor their own progress. 
If students had problems with a particular topic, their list 
could be used as a focus to identify where they are in their 
studies and what they need to concentrate attention on. 

The students were given guidelines of the attainment level 
expected from them by the end of the unit. This differed 
for each topic, an example would be "achieve all items in 
the Medium category and a third of the items in the High 
category". So providing students cover all the items to the 
required minimum level (i.e.. Medium), they could select 
which items to concentrate on for in-depth study. Further 
refinement in the guidance could be given, for instance 
stipulating core items that students are expected to 
achieve a high level of knowledge in, however, this was 
not used on this unit, the intention being to get the 
students to have responsibility for their own study which 
included identifying which topics to focus on. 

The unit assessment was designed to recognise and 
complement this self selection of topics. The assessment 
consisted of students researching into their own selected 
topic area (within the confines of the unit), and involved 
producing a written report and a presentation to the rest of 
the class. In addition, a case study assessment was used at 
the end of the unit in which a range of issues from each 
topic would be relevant, enabling students to demonstrate 
and apply their in-depth learning in their selected items. 

Students were encouraged to review their own progress by 
examining their own list at the end of the topic. This way 
they could see the 'value addedness' of their study. 

Feedback from students on the use of this technique was 
generally very positive. Some of the comments include "it 
helps to know what to read about" and "helps to 
concentrate study". It also seemed to help to reduce the 
alienation of students with some topic areas. For instance, 
in topics where many students recorded their knowledge 
as very low for most of the items, it proved useful to see 
that their knowledge profiles for the topic was not too 
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FIGURE 1 

SKILLS AUDIT FOR LEGAL ASPECTS 



Level of Knowledge/ 
understanding 


Item (please add your own description) 


Very low 


Medium 


High 


Data Protection Act (general details of) 








Data Protection Act - eight principles 








Changes in Data Protection legislation for EC Directive for October 
1998 








FAST 








BSA 








Legal Aspects of using E-commerce 








Searching for Legal information (past and current acts of Parliament) 








Negligence Law relevant to IT 








Contract Law relevant to IT 








Legal issues of the Year 2000 'bug* 








Computer Misuses act 








Legal responsibilities for people using IT 








Legal issues covering access to 'Outsourced* source code (if someone 
provides you software do you have a legal right to access the source code?) 








In a computing environment, what items can be covered by insurance? 








Warranties 








Intellectual property rights 









dissimilar to others and to promote discuss of what level 
of knowledge they should be aiming for in the topic. 
Overall, I found the technique very powerful in 
encouraging students to take responsibility for their own 
study. 

DISCUSSION: THE PEDAGOGY OF SKILLS 
AUDIT 

Changes, debate and writings on assessment issues in 
higher education has a long history (Tillyard 1913; 
Winter 1993, p365); change in education is not new and 



is an evolutionary process. Much of the recent literature 
deals with how to maintain quality in the assessment 
process while student numbers increase and resources are 
reduced (Gibbs, Lucas and Simonite 1996, p261). Many 
of these works examine innovative assessments such as 
self assessment and peer assessment (Gibbs 1989; Gibbs 
and Jenkins 1992; Hughes and Large 1993; Boud 1981, 
1986; Adams and King 1993; Cheng and Warren 1997). 
The debate over assessment is very current and is likely to 
continue with the impending changes in higher education 
(Jacobs 1 997). The debate is further enhanced with the 
advances in technology making distance learning more 
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attractive, and raises several questions on how to assess 
from a far, both summatively and formatively (Minoli 
1997; Kapur and Stillman 1997). The resulting increased 
variety of entry routes, with their corresponding diversity 
of skill-sets, highlight the need for a monitoring device for 
the changing skill-set on new entrants. An initial skills 
audit, such as described in the Business Skills case in this 
paper, is one such example. 

Probably the most common connotation of auditing is the 
accountancy sense of auditing a company’s financial 
information (e.g. BPP 1997). A wider definition would 
include such items as *to examine’, ’to verify', ’to take 
stock*, ’to check*. Defining auditing within a particular 
area will be very context related, for instance, audit in a 
computing sense would include such items as *check on 
the accuracy and completeness of the results of computer 
processing* and 'check how efficiently a computer is being 
used’ (Galland 1983) or more currently, checking on a 
range of items such as software licences and access 
controls (Alter 1996). 

A key aspect of auditing from a student skills perspective 
would be the ’taking stock* of student knowledge. This 
already takes place in most institutions, typically before 
a course starts or at the end of a course. 'Taking stock* of 
student skills at the start of a course will include setting 
and checking pre-requisite entry qualifications, so that 
faculty can expect a minimum level of skills from the 
student cohort. 'Taking stock* at the end of a course (e.g. 
an exam) gives an indication that students have met the 
learning and attainment criteria for that course. The 
results of this taking stock is used by faculty, students and 
others. What is less widely used is the 'taking stock* 
during the course and, the use of that 'taking stock* by the 
students during the course. This sort of auditing of student 
skills is not new and has been used successfully 
elsewhere, a good example being the Maths Science 
Inventory (LeBold, Budny and Ward 1998). 

As can be seen, some sort of skills audit is already widely 
used in higher education, usually at the start and end of a 
course. However, less commonly used is the self student 
audits during the course. The self skills audit can be 
considered as a Self Assessment Skill Activity (SASA) 
(Adams and King 1994) that can be used to develop 
student learning patterns especially if a systematic 
approach is taken (Boud 1986, p27). Though the skills 
audit for the level 3 unit described in this paper does not 
form part of the formal (summative) assessment, it would 
only take a small step to make it so. A self audit may be 
an excellent start for a learning contract (Sutherland 
1997, pl93; Brown and Knight 1994, p90). As such a 



skills audit could meet the summative^ i.e., giving a mark, 
mdformative, i.e., giving feedback, aspects of assessment 
and encourage more learning independence in the 
students. 

A self skills audit seems particularly relevant for distance 
learners. As has been discussed elsewhere, distance 
learners have a unique set of assessment needs and 
associated problems (Harris and Bell 1994, pi 64; Minoli 
1996), particularly when it comes to feedback and contact 
with other students and staff. A self assessment of own 
skills would go some way to meet the feedback needs of 
distance learners. It may also be used as a focus for 
discussion between students and between staff and 
students, especially if used in conjunction with technology 
(Kapur and Stillman 1997), such as in providing 
discussion notice boards, possibly containing group results 
of self assessments. 

SUMMARY 

The use of skills audit cases described in this paper shows 
a progression from level 1 to level 3 of undergraduate 
study. At level 1 , the main aims were to examine the skills 
of the new entrants enabling early identification of weaker 
students, quick feedback to the students and, monitoring 
the delivery of topics within the unit. At level 3, the main 
aims were to foster self study, changing the emphasis on 
what the tutors do to what the students do. The use has 
changed from predominantly a monitoring tool for staff at 
level for level 1 to a study tool for students at level 3. 
Level 1 is used as mainly formative assessment, level 3 as 
mainly summative assessment. Ownership of the audit 
moves from the tutor in level 1 towards the student at 
level 3. A self audit seems particularly appropriate for 
distance learners, especially if used in conjunction with 
distance learning technology. 

Overall, the skills audit approach is a simple yet powerful 
tool in the teaching and learning process. Given the 
changes in the UK education system, a skills audit has use 
as a monitoring device for the changing skill-set on new 
entrants. The changes in education results in a broader 
skill-set of a cohort of students along with reduced 
feedback to individual student, i.e., a reduction in 
formative feedback. It is increasingly difficult in the IS/IT 
field to monitor individual student performance and give 
individual feedback and direction. The use of a self skills 
audit provides one mechanism to meet the formative 
feedback needs of students. This seems particularly 
appropriate for distance learning. 
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